TRECVID 2005 Experiments at Johns Hopkins University: Using Hidden Markov Models for Video Retrieval

The use of hidden Markov models (HMMs) for the high-level feature detection task of TRECVID 2005 is described. Highlevel features present in a keyframe are assumed to constitute the state-space of a Markov chain. The observed visual features of the keyframe, as well as the text accompanying the keyframe, are modeled as stochastic emissions from (unobserved) states of this Markov chain. Manual annotations of shots for the presence of these high-level features, as provided by NIST, constitute the data from which parameters of the HMM are estimated. The estimated HMM enables computation of the a posteriori probability that a particular high-level feature is present in a keyframe in the test collection, given the visual features of the keyframe and its accompanying closed-caption or transcribed text. It is demonstrated that different subsets of the set of visual and text features, and slightly different HMM-settings, are optimal for detecting different high-level features. Finally, it is demonstrated that the posterior probabilities of highlevel features in a test keyframe, computed by the HMMs, may themselves be used as inputs to a support vector machine to further improve detection performance. Results for high-level feature detection are presented on a held-out portion of the manually annotated NIST corpus, as well as the TRECVID 2005 video search collection.